' A THREE-DIMENSIONAL DATA INPUT METHOD AND APPARATUS 

This application is based on Japanese Patent 
Application No. 87552/1999 filed on March 30, 1999 and No. 
5 87553/1999 filed on March 30, 1999, the contents of which 
are incorporated herein by reference. 



BACKGROUND OF THE INVENTION 
Field of the Invention 

10 The present invention relates to a method and an 

apparatus for inputting three-dimensional data of an object. 
Description of the Prior Art 

An optical three-dimensional data input apparatus (a 
three-dimensional camera), which enables rapid measurement 

15 compared with a contact type, is used for data input into a 
CG system or a CAD system, physical measurement, visual 
sense of a robot or other applications. A slit light 
projection method (also referred to as a light cutting 
method) is known as the measurement method suitable for this 

20 three-dimensional data input apparatus. 

Fig. 13 shows an input principle of a three-dimensional 
camera 80 utilizing the slit light projection method. 

In Fig. 13, a three-dimensional camera 80 includes a 
light projection portion 81 and a light reception portion 82. 

25 The light projection portion 81 irradiates a slit light 
beam S having a linear section. The light reception 
portion 82 has an image sensing surface 83 and an imaging 
lens (not shown). The light projection portion 81 and the 
light reception portion 82 are separated from each other by 

30 a predetermined distance and are integrally installed in 
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oiie* housing. 

The object Ql is irradiated by the slit light beam S 
from the light projection portion 81, and the reflected 
light is captured on the image sensing surface 83 as a slit 

5 image. A space coordinate of the point p on the object Ql 
corresponding to a point p' of this slit image is 
determined as the coordinates of the intersection point of 
the plane formed by the slit light beam S and the line L 
connecting the point p' and the center point 0 of the 

10 imaging lens. Therefore, the space coordinates of points 
on the surface of the object Ql corresponding to the slit 
image are derived from one slit image obtained by the slit 
light beam S. The slit light beam S is moved horizontally 
so as to scan the object Ql and the slit image at each 

15 scanning position is input. Thus, the three-dimensional 

data (three-dimensional shape data) of the front portion of 
the object Ql are inputted, which is irradiated by the slit 
light beam S. 

In order to obtain the three-dimensional data of the 
20 all-around surface of the object Ql, it is necessary to 

input from plural directions of the object Ql. Two methods 
are known for the purpose. In the first method, the three- 
dimensional camera 80 is moved on a predetermined track 
around the object Ql while the shooting direction of the 
25 camera is directed to the object Ql so as to take images of 
the object Ql from plural directions. In the second method, 
a rotation stage is used on which the object Ql is placed. 

The three-dimensional camera 80 is placed at a 
predetermined position for taking images of the object Ql 
30 from plural directions. 
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' The plural sets of three-dimensional data of the object 
Ql taken from plural directions are processed for 
registration using a conversion parameter derived from a 
position on the track along which the three-dimensional 

5 camera 80 moves or the position of the rotation stage. Thus, 
the three-dimensional data of the all-around surface of the 
object Ql are obtained. 

However, the above-mentioned method causes a high cost 
since the position of the three-dimensional camera 80 or the 

10 angular position of the rotation stage should be detected 
in high accuracy for improving the registration accuracy. 

In addition, since the three-dimensional camera 80 
should be placed on a moving device, it is impossible to 
take images with holding the three-dimensional camera 80 in 

15 hand. Accordingly, the object that can be inputted is 
limited. Namely, the object such as a stone or bronze 
statue that cannot be moved cannot be inputted as the 
three-dimensional data by this method. 

An apparatus that can solve this problem is disclosed 

20 in US Patent No. 5, 990, 895. This apparatus can take images 
of the object Ql from any direction for inputting three- 
dimensional data. After inputting, three-dimensional 
shapes inputted from plural directions as well as a color 
image inputted simultaneously from the same view and 

25 corresponding to the three-dimensional data are displayed. 
A user designates corresponding points in the three- 
dimensional shape manually viewing the displayed color image, 
in accordance with a variation of color or other factors. 
The registration among the inputted three-dimensional data 

30 is performed on the basis of the corresponding points 
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designated by the user. 

However, according to the above-mentioned conventional 
apparatus, the user has to designate corresponding points 
for the registration among plural sets of three-dimensional 

5 data. This process is extremely troublesome. 

Furthermore, there is no method for confirming whether 
proper images have been taken or not for all-around surface 
of the object Ql when three-dimensional data input is 
performed. Accordingly, if the plural three-dimensional 

10 data are not continuous having proper overlaps, or if there 
is some lack of data, it is difficult to make proper 
corresponding points between three-dimensional data. In 
that case, accuracy of the registration between the three- 
dimensional data may be deteriorated. In the worst case, 

15 it is necessary to take images again. 

SUMMARY OF THE INVENTION 
The object of the present invention is to provide a 
method and an apparatus that can input three-dimensional 
20 data of an object easily, quickly and correctly, and enables 
to obtain a composite image of the object for all-around 
surface or a predetermined area without any lack. 

According to a first aspect of the present invention, a 
three-dimensional data input method is provided, which uses 
25 a three-dimensional data input apparatus that is 

constructed to input the three-dimensional data of an object 
displayed on a monitor screen by shooting the object. The 
three-dimensional data input method comprises the steps of 
generating image data of the three-dimensional shape model 
30 in accordance with the three-dimensional data inputted from 
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a 'part of the object, the image corresponding to the shape 
of the three-dimensional data; displaying the image of the 
three-dimensional shape model on the monitor screen as a 
guide image for framing; performing the framing so that the 

5 guide image is overlapped on the image of the object image 
that corresponds to the guide image; and shooting the object 
after the framing performed. 

According to a second aspect of the present invention, 
a three-dimensional data input apparatus is provided, which 

10 is used for obtaining three-dimensional data of an object by 
shooting the object. The three-dimensional data input 
apparatus comprises a monitor for displaying the object; an 
image generator for generating a three-dimensional model 
image of the object in accordance with the three- 

15 dimensional data of the object obtained by the shooting; and 
a display controller for displaying the three-dimensional 
model image as a guide image on the monitor for framing. 

According to a third aspect of the present invention, a 
three-dimensional data input method is provided, which uses 

20 a three-dimensional data input apparatus that includes a 

monitor screen for confirming an object and is constructed 
to input the three-dimensional data of the object by 
shooting the object. The three-dimensional data input 
method comprises the steps of displaying an image of a 

25 three-dimensional shape model having a shape that is 

substantially identical to the shape of the object as a 
guide image for framing on the monitor screen; framing in 
accordance with the guide image; and shooting the object 
after the framing is performed. 

30 According to a fourth aspect of the present invention, 
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a 'three-dimensional data input method is provided, which 
uses a three-dimensional data input apparatus that includes 
a monitor screen for confirming an object and is 
constructed to input the three-dimensional data of the 
object by shooting the object. The three-dimensional data 
input method comprises the steps of memorizing an attribute 
information in a memory, the attribute information being 
about data of a three-dimensional shape model having a 
shape that is substantially the same as the shape of the 
object and a position for observing the three-dimensional 
shape model; displaying the image of the three-dimensional 
shape model as a guide image for framing on the monitor 
screen in accordance with the attribute information; and 
framing so that the image of the input portion is 
overlapped on the guide image for shooting the object after 
the framing is performed. 

According to a fifth aspect of the present invention, a 
three-dimensional data input apparatus is provided, which 
is used for obtaining three-dimensional data of an object 
by shooting the object. The three-dimensional data input 
apparatus comprises a monitor for displaying the object; a 
first memory for memorizing an attribute information about 
data of a three-dimensional shape model having a shape that 
is similar to the shape of the object and a position for 
observing the three-dimensional shape model; and a display 
controller for displaying the image of the three- 
dimensional shape model as a guide image for scaling on the 
monitor in accordance with the attribute information. 

BRIEF DESCRIPTION OF THE DRAWINGS 
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1 ■ Fig. 1 is a diagram showing a configuration of a 
measurement system utilizing a three-dimensional camera 
according to the present invention. 

Figs. 2A and 2B are diagrams showing an appearance of 
5 the three-dimensional camera. 

Fig. 3 is a block diagram showing a functional 
configuration of the three-dimensional camera. 

Figs. 4A and 4B are diagrams showing a concrete method 
of inputting three-dimensional data of all-around surface or 
10 a predetermined area of an object using the three- 
dimensional camera. 

Fig. 5 is a diagram for explaining a process of 
generating a monitor image and a guide image by an 
overlapping image generating portion according to a first 
15 embodiment of the present invention. 

Figs. 6A-6C are diagrams showing a monitor image and a 
guide image of an object displayed on a finder according to 
the first embodiment of the present invention. 

Fig. 7 is a flowchart showing an operation and a 
20 process of inputting the three-dimensional data according to 
the first embodiment of the present invention. 

Fig. 8 is a diagram for explaining a process of 
generating a monitor image and a guide image by an 
overlapping image generating portion according to a second 
25 embodiment of the present invention. 

Fig. 9 is a diagram showing a monitor image and a guide 
image of an object displayed on a finder according to the 
second embodiment of the present invention. 

Fig. 10 is a flowchart showing an operation and a 
30 process of inputting the three-dimensional data in a first 
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irtputting method according to the second embodiment of the 
present invention. 

Fig. 11 is a diagram showing a monitor image and a 
guide image of an object displayed on a finder in a second 
inputting method according to the second embodiment of the 
present invention. 

Fig. 12 is a flowchart showing an operation and a 
process of inputting the three-dimensional data in the 
second inputting method according to the second embodiment 
of the present invention. 

Fig. 13 a diagram showing an input principle of a 
three-dimensional camera utilizing the slit light 
projection method. 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 
Fig. 1 is a diagram showing a configuration of a 
measurement system 1 utilizing a three-dimensional camera 2 
according to the present invention. Figs. 2A and 2B are 
diagrams showing an appearance of the three-dimensional 
camera 2. 

As shown in Fig. 1, the measurement system 1 includes a 
three-dimensional camera 2 and a host 3. 

The three-dimensional camera 2 is a portable type 
three-dimensional data input apparatus for three- 
dimensional measurement utilizing the slit light beam 
projection method, which inputs three-dimensional data 
(distance data) by taking images of the object Q to be 
inputted, and calculates and outputs data that are a basis 
of determining the three-dimensional shape of the object Q 
in accordance with the three-dimensional data. 
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' ' The host 3 is a computer system including a CPU 3a, a 
display 3b, a keyboard 3c and a mouse 3d. Data can be 
transferred by online or a removable recording medium 4 
between the three-dimensional camera 2 and the host 3. The 

5 recording medium 4 includes a magneto-optic disk (MO), a 
mini disk (MD) and a memory card. 

The host 3 performs an operational process calculating 
coordinates of sampling points using triangulat ion 
techniques on the basis of the three-dimensional data given 

10 by the three-dimensional camera 2 and a paste process 
(combination process). Software for the processes is 
installed in the CPU 3a. 

As shown in Fig. 2A, the three-dimensional camera 2 has 
a light irradiation window 20a and a light reception window 

15 20b on the front surface of a housing 20. The light 

irradiation window 20a is arranged at the upper portion of 
the light reception window 20b. A slit light beam (a laser 
beam having a slit-like section with predetermined width w) 
U is emitted by an optical unit OU inside the housing 20 

20 and passes through the light irradiation window 20a toward 
the object Q. The radiation angle <j> of the slit light 
beam U in the longitudinal direction Ml is fixed. A part 
of the slit light beam U reflected by the surface of the 
object Q passes through the light reception window 20b and 

25 enters the optical unit OU. The optical unit OU has a 
biaxial adjustment mechanism for adjusting relationship 
between the light irradiation axis and the light reception 
axis. 

The upper surface of the housing 20 is provided with 
30 zooming buttons 25a, 25b, manual focusing button 26a, 26b, 
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and' reception that make up the above-mentioned optical unit 
OU. In the optical system 40, a semiconductor laser (LD) 41 
emits a laser beam having wavelength of 685 nm, which 
passes through a light irradiation lens system 42 and 

5 becomes the slit light beam U to be deflected by a 

galvanomirror (scanning device) 43. A driver 44 of the 
semiconductor laser 41, a drive system 45 of the light 
irradiation lens system 42 and a drive system 46 of the 
galvanomirror 43 are controlled by a system controller 61. 

10 In the optical system 50, the light beam condensed by a 

zooming unit 51 is divided by a beam splitter 52. 

The light beam having a wavelength within the 
oscillation wavelength band of the semiconductor laser 41 
enters a measurement sensor 53. A light beam having a 

15 wavelength within the visible band enters a monitoring color 
sensor 54. The measurement sensor 53 and the monitoring 
color sensor 54 are CCD area sensors. The measurement 
sensor 53 and the monitoring color sensor 54 output 
shooting information of the object Q and image information, 

20 respectively. 

The zooming unit 51 has a zoom lens (not shown). This 
zoom lens is moved between the minimum focal length point 
and the maximum focal length point along the shooting 
direction, so that the three-dimensional data can be 

25 inputted in various resolution values. Furthermore, a part 
of the incident light beam is used for auto focusing (AF). 

The AF function is achieved by an AF sensor 57, a lens 
controller 58 and a focusing drive system 59. A zooming 
drive system 60 is provided for electric zooming. 

30 Next, principal flows of electric signals in the three- 
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arid' a shutter button 27. 

Furthermore, as shown in Fig. 2B, the rear surface of 
the housing 20 is provided with a finder 21, a cursor 
button 22, a select button 23, a cancel button 24, an 

5 analog output terminal 32, a digital output terminal 33 and 
a slot 30a for a recording medium 4. 

The finder 21 is an image display device having a 
monitor screen such as a liquid crystal display. On the 
finder 21, a monitor image QP and a guide image GP are 

10 displayed as a feature of the present invention. The 

monitor image QP and the guide image GP will be explained 
in detail later. Furthermore, operation procedure 
information that gives indications to the user what to do 
the next in each operation step using characters and marks 

15 is displayed. In addition, a distance image (a light and 
shade image) is displayed, which express the three- 
dimensional data of the shot portion by light and shade. 

The buttons 22-24 on the rear surface are used for 
setting shooting mode or others. The analog output terminal 

20 32 outputs a two-dimensional image signal of the object Q 

in the format such as NTSC. The digital output terminal 33 
is a SCSI terminal, for example. 

Next, the function of the three-dimensional camera 2 
will be explained. 

25 Fig. 3 is a block diagram showing a functional 

configuration of the three-dimensional camera 2. Full line 
arrows indicate flows of electric signals and broken line 
arrows indicate flows of light in Fig. 3. 

As shown in Fig. 3, the three-dimensional camera 2 

30 includes two optical systems 40, 50 for light irradiation 
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1 ' dimensional camera 2 will be explained. 

The shooting information of the measurement sensor 53 
is transferred to an output process circuit 62 in 
synchronization with a clock signal given by the driver 55. 

5 The output process circuit 62 includes an amplifier for 

amplifying a photoelectric conversion signal of each pixel 
outputted by the measurement sensor 53 and an A/D 
conversion portion for converting the photoelectric 
conversion signal into 8-bit received light signal. The 

10 received light signal obtained by the output process circuit 
62 is temporarily memorized by a memory 63 and is 
transferred to a barycenter operation circuit 73. The 
address assignment is performed by a memory control circuit 
63A. The barycenter operation circuit 73 calculates data to 

15 be a basis for calculating a three-dimensional shape in 

accordance with the received light signal that was inputted, 
which is outputted to an output memory 64. Furthermore, the 
barycenter operation circuit 73 generates a distance image 
that corresponds to the shape of the object Q and outputs it 

20 to a display memory 74. The data stored in the output 

memory 64 are outputted from the digital output terminal 33 
via a SCSI controller 66, or are outputted to a recording 
medium 4. The distance image stored in the display memory 
74 is displayed on the finder 21 via a display controller 

25 76. 

The imaging information obtained by the monitoring 
color sensor 54 is transferred to a color process circuit 67 
in synchronization with a clock signal given by the driver 
56. After the color process, the imaging information is 
30 outputted online via an NTSC conversion circuit 70 and the 
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' ' an'alog output terminal 32, or is stored in the color image 
memory 69 after digitized by a digital image generator 68. 

The imaging information stored in the color image memory 69 
is output online from the digital output terminal 33 via 

5 the SCSI controller 66, or is written into the recording 
medium 4. Furthermore, the imaging information is 
displayed on the finder 21 as a monitor image QP via the 
display controller 76. 

The system controller 61 gives instructions to the 

10 display controller 76 for displaying operation procedure 

information on the finder 21. A model data memory 75, the 
color image memory 69, the display controller 76, the system 
controller 61 and the finder 21 make up the overlapping 
image generating portion 7. 

15 Using the three-dimensional camera 2 having the above- 

mentioned configuration and functions, three-dimensional 
data of the object Q for all-around surface or a 
predetermined area can be inputted. 

Figs. 4A and 4B are diagrams showing a concrete method 

20 of inputting three-dimensional data of all-around surface or 
a predetermined area of an object Q using the three- 
dimensional camera 2. 

In the method shown in Fig. 4A, the user moves the 
three-dimensional camera 2 around the object Q for inputting 

25 after changing the shooting condition. In the method shown 
in Fig. 4B, the object Q is moved for inputting after 
changing the shooting condition. In addition, the three- 
dimensional camera 2 can be attached to a predetermined arm, 
which is moved so as to change the position for shooting 

30 the object Q. 
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• Next, the monitor image QP and the guide image GP in a 
first embodiment of the present invention will be explained 
with reference to Fig. 5 and Figs. 6A-6C. 

Fig. 5 is a diagram for explaining a process of 

5 generating the monitor image QP and the guide image GP by 

an overlapping image generating portion 7. Figs. 6A-6C are 
diagrams showing the monitor image QP and the guide image 
GP of an object Q displayed on a finder 21. 

In Fig. 5 and Figs. 6A-6C, the monitor image QP is a 

10 color image for monitoring the portion of the object Q to 
which the user directs the view line of the three- 
dimensional camera 2 for inputting three-dimensional data, 
and is generated from the imaging information stored in the 
color image memory 69. 

15 The guide image GP gives information to the user 

position and how the object Q should be shot for inputting 
three-dimensional data of all-around surface or a 
predetermined area of the object Q. The guide image GP is 
generated from object shape data TD as shown in Fig. 5. 

20 The object shape data TD are three-dimensional data of 

the input portion of the object Q. In the present 
embodiment, the left side body of a rabbit is illustrated 
as an example. 

The object shape data TD are processed such as wire 

25 frame, shading, texture mapping or coloring. The coloring 
process is performed for easy discrimination of the monitor 
image QP from the image processed by wire frame, shading, 
texture mapping or coloring. The coloring process generates 
blue, green, red or other color, for example. Furthermore, 

30 the ratio of densities of two overlapping images can be 
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' 1 ch'anged to any value by changing mixing ratio of the 

monitor image QP and the guide image GP so that the user can 
observe easiest. These processes can facilitate 
overlapping of the guide image GP and the monitor image QP. 

5 The process can be selected in accordance with user' s 
desire. In the present embodiment, an example is shown 
with processed by texture mapping. 

The user can rotate (change the posture of) the object 
shape data TD or zoom the same so that the guide image GP 

10 becomes the state viewed from a desired direction. Thus, 
the guide image GP is changed. The user can determine the 
next shooting position with reference to this guide image 
GP. Hereinafter, if it is necessary to distinguish the 
guide image GP from others by its displayed shape, a suffix 

15 number is added in such way as "GP1" or "GP2. " 

The method for inputting the three-dimensional data 
using the monitor image QP and the guide image GP will be 
explained with reference to Fig. 5 and Figs. 6A-6C. 

First, the user takes an image of the object Q from one 

20 direction. For example, the left half body of the rabbit 
is shot from the right direction. The three-dimensional 
data of the left half body of the rabbit are obtained by the 
shooting. The obtained three-dimensional data are 
displayed on the finder 21 as the guide image GP1 as shown 

25 in Fig. 6A. 

Next, the user rotates the guide image GP1 (object 
shape data TD) in the right direction viewed from the above, 
for example, so that the guide image GP2 that is the state 
of the object viewed from the rear side as shown in Fig. 6B. 

30 On this occasion, the right edge area GPR of the object 
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shape data TD is displayed on the finder 21. Thus, an 
overlapping area is secured for combining the object shape 
data TD obtained by the first shooting and the object shape 
data TD obtained by the second shooting. If a color of the 

5 edge area to be the overlapping area is different from the 
other area, the user can observe easily. 

Next, as shown in Fig. 6C, framing is performed so that 
the guide image GP2 overlaps the portion of the monitor 
image QP corresponding to the guide image GP2. 

10 As shown in Fig. 4, either the three-dimensional camera 

2 or the object Q can be moved for framing. By performing 
zooming or by moving the three-dimensional camera 2, various 
resolutions can be used for inputting. For example, a high 
resolution is used for inputting detail data required to a 

15 portion such as a face, while a low resolution can be used 
for inputting a portion such as a back side whose shape is 
not changed so rapidly. Thus, unnecessary increase of data 
can be avoided. 

After framing, a shutter button 27 is pushed for the 

20 second shooting. In the second shooting, the rabbit is shot 
at the back side. 

By the similar operation, the guide image GP2 is 
changed sequentially to the guide image GP3, GP4, . . . (not 
shown), so that the all-around surface of the object is 

25 shot. Thus, three-dimensional data of the object for all- 
around surface are inputted. 

The generation of the guide image GP3, GP4, . . . from 
the third time is performed by the following method, for 
example. 

30 (1) The guide image GP is generated from the image that 
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' ' wa's • obtained by the last shooting. 

(2) The guide image GP is generated by using all images 
obtained by the previous shooting. 

(3) The guide image GP is generated by using any image 
5 selected from all images obtained by the previous shooting. 

Thus, the sequential change of the guide image GP can 
be performed by a manual mode performed in accordance with 
the user' s desire or by an automatic mode performed by the 
program in accordance with a predetermined order. If the 

1° manual. mode is selected, a mouse or other device can be 
used for operation. 

With reference to the guide image GP, the three- 
dimensional data of the object can be inputted easily, 
quickly and correctly. Furthermore, registration of the 

15 plural obtained three-dimensional data can be performed 

easily with high accuracy, so that a composed image of all- 
around surface or a predetermined area of the object can be 
obtained easily without omission. 

Fig. 7 is a flowchart showing an operation and a 

20 process of inputting the three-dimensional data. 

The object is shot from an appropriate position as a 
first view point, so that the first three-dimensional data 
are inputted (#1). A finder display view point is 
determined in the coordinate system upon inputting the first 

25 three-dimensional data (#2). A guide image GP1 is 

displayed in accordance with the first three-dimensional 
data at the first view point (#3). The guide image GP1 
becomes a guide image GP2 by automatic or user designated 
rotation and zooming. The guide image GP2 is overlapped on 

30 the monitor image QP from the next view point, i. e. , the 
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se'c6nd view point (#4). After the overlapping is succeeded, 
the second three-dimensional data are inputted (#5). A 
parameter for converting the second three-dimensional data 
into the coordinate system of the first three-dimensional 

5 data is determined. The three-dimensional data overlapped 
on the finder 21 for inputting are data of each coordinate 
system. The conversion is required between these coordinate 
systems. The registration of the three-dimensional data 
means a conversion into one of the coordinate systems. Two 

10 methods for the registration can be used as explained below 
(#6). A coordinate conversion is performed to the second 
three-dimensional data using the determined parameter (#7) 
and the initial registration is completed (#8). A fine 
adjustment is performed by ICP process for further 

15 registration with high accuracy (#9). 

The ICP is a method of minimizing a total distance 
between corresponding points by making the nearest point the 
corresponding point. The detail of the ICP is explained in 
M A method of registration of 3-D shapes" (P. Besl and N. 

20 McKay, IEEE Transactions on Pattern Analysis and Machine 
Intelligence, 12(2): 239-256, 1992). 

Next, the two methods for registration mentioned in 
(#6) will be explained. 
[The first registration method] 

25 The following explanation uses the parameters below. 

Pt: view point position when generating the guide image 

GP 

Vt: view line direction when generating the guide image 

GP 

30 Ut: upward vector when generating the guide image GP 
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' Ft: focal distance when generating the guide image GP 
St: distance to the noted point when generating the 
guide image GP 

Fa: focal distance when shooting 
5 Sa : shooting distance 

Pa: view point position when shooting 
Va: view line direction when shooting 
Ua: upward vector when shooting 

Pa' : view point position after coordinate conversion 
10 Va' : view line direction after coordinate conversion 

Ua' : upward vector after coordinate conversion 
In the method explained here, the view point position 
Pa when shooting is determined for the registration. When 
generating the guide image GP displayed on the finder 21, 
15 the view point position Pt and the view line direction Vt 
are set. 

The view line direction Va and the view point position 
Pa when shooting the object Q while overlapping the monitor 
image QP on the guide image GP on the finder 21 are as 

20 follows. Namely, the view line direction Va is the 

direction same as the view line direction Vt (see the 
expression (3) below), and the view point position Pa is 
moved forward and backward in the same direction as the view 
line direction Vt. 

25 Since the lens information (focal length) at shooting 

is obtained, the shooting distance Sa is determined so that 
the image scaling ratio becomes the same as the guide image 
GP. Thus, the moving distance (St - Sa) along the view 
line direction Va (= Vt) can be calculated and the view 

30 point position Pa can be derived as the following equation 
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(4')/ Namely, since 

Ft/St = Fa/Sa (1), 

the shooting distance Sa is derived from the following 
equation (2) : 

5 Sa = (Fax St)/Ft (2). 

Furthermore, since 

Va = Vt and Ua = Ut (3), 

the view point position Pa at shooting is derived from the 
following equation (4): 

10 Pa - Pt + (St - Sa) x Vt (4). 

The three-dimensional data obtained from each direction 
are moved in parallel in accordance with the view point 
position Pa and are rotated in accordance with the view 
line direction Va, so as to convert it into the coordinate 
15 system of the original three-dimensional data used for 
generating the guide image GP. The parallel movement 
quantity T for the coordinate conversion into the 
coordinate system of the original three-dimensional data is 
derived from the following equation (5). 

20 T = Pa' - Pa (5) 

The view line direction Va' after the coordinate 
conversion and the upward vector Ua' after the coordinate 
conversion can be expressed in the equations (6) and (7) 
using Va and Ua and a rotation matrix R. 

25 Va' = R x Va (6) 

Ua' = R x Ua (7) 

The three-dimensional data obtained by the parallel 
movement quantity T and the rotation matrix R from each 
direction are converted into the data of the coordinate 
30 system of the original three-dimensional data so as to 
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perform the registration. 

Overlapping operation generates the relative movement 
of the three-dimensional camera 2 to the object Q. The 
three-dimensional data inputted from plural positions are 
converted into the data of the coordinate system of the 
first shooting, in accordance with the movement. Thus, the 
registration is performed. 

As explained above, the three-dimensional data inputted 
from plural positions are inputted with high accuracy of 
registration by overlapping the monitor image QP on the 
guide image GP. Therefore, a post registration is not 
required. 

Thus, the host 3 performs pasting process on the basis 
of the three-dimensional data from plural positions with 
high accuracy of registration. Therefore, the pasting 
process in the host 3 can be performed in high speed and 
high accuracy. 

[The second registration method] 

In this method, the registration is achieved utilizing 
the fact that the overlapping portion of the three- 
dimensional data is known. 

The following explanation uses the parameters below. 

Pti: corresponding points of data that were shot first 

Pai: corresponding points of data that was shot after 
registration on the finder 

T: parallel movement quantity 

R: rotation matrix indicating rotation movement 
Concerning points in the overlapping area on the finder 
21, the nearest point on this two-dimensional image is the 
corresponding point. The registration is performed by 



2 1 



10 



15 



20 



25 



30 



coordinate conversion such that the distance of the 
corresponding point on three-dimension becomes the minimum. 

The coordinate conversion that minimizes the distance of 
the corresponding point can be obtained by determining the 
parallel movement quantity so that the barycenter positions 
of the corresponding points are identical, and by obtaining 
the rotation movement using the least-squares method. The 
following equations (8) and (9) is used. 



T = 



ZPti - ZPai 



N 



(8) 



The rotation matrix R is determined so as to minimize J. 

2 



J = Z 



Pti 



R x (Pai - T ) 



.. (9) 



The detail of this calculation method is explained in 
"The representation, recognition and locating of 3-D 
objects, " Faugeras and Hebert, 1986 0. Faugeras and M. 
Hebert, International Journal of Robotics Research 5(3): 
27-52, 1986. 

The coordinate conversion of each point Pi of the 
second three-dimensional data is performed using the 
parallel movement quantity T and the rotation matrix R 
obtained as mentioned above, on the basis of the following 
equation (10). 

P' i = Rx (Pi - T) (10) 

The above-explained coordinate conversion enables the 
registration of the first three-dimensional data and the 
second three-dimensional data. 

In the above-explained embodiment, the guide image GP 
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1 ' and 'the monitor image QP are displayed on the finder 21 
that is integrated to the three-dimensional camera 2. 
However, they can be displayed on a finder separated from 
the three-dimensional camera 2 or on a display 3b of the 
5 host 3. 

In the above-explained embodiment, a see-through 
display device utilizing a half mirror can be used as the 
finder 21. In this case, power consumption of the three- 
dimensional camera 2 can be reduced. 

10 According to the above-explained embodiment, the guide 

image GP is generated on the basis of the three-dimensional 
data of the shot object Q, so the effort of premaking the 
guide image GP can be saved and the memory for storing the 
guide image GP can be minimized. 

15 Next, the monitor image QP and the guide image GP in a 

second embodiment of the present invention will be explained 
with reference to Fig. 8 and Figs. 9A-9C. The rabbit as 
the object in the first embodiment is replaced with a head 
portion of a human as the object Q in this embodiment. 

20 The guide image GP gives the user the information about 

from which position and how the object Q should be shot for 
inputting three-dimensional data of all-around surface or a 
predetermined area of the object Q. Such information is 
generated from model data MD and attribute information AJ as 

25 shown in Fig. 8. 

The model data MD is three-dimensional shape model data 
having substantially the same shape as the object to be 
inputted. The model data MD a standard shape model of each 
of plural kinds such as a head of human (including a face), 

30 a human body, an animal, a car, a statue, an earthenware or 
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a bell-shaped bronze vessel, whose shape is known and can 
be standardized. The model data MD are stored in the model 
data memory 75 by each kind. 

The user can select the model data MD in accordance 

5 with the kind of the object Q to be inputted by a 

predetermined operation. Data generated from the three- 
dimensional data inputted beforehand by the three- 
dimensional camera 2 or data produced by CG technique on a 
computer can be used as the model data MD. 

10 The model data MD are processed such as wire frame, 

shading, texture mapping or coloring. The coloring process 
is performed for easy discrimination of the monitor image 
QP from the image processed by wire frame, shading, texture 
mapping or coloring. The coloring process generates blue, 

15 green, red or other color, for example. Furthermore, the 

ratio of densities of two overlapping images can be changed 
to any value by changing mixing ratio of the monitor image 
QP and the guide image GP so that the user can observe 
easiest. These processes can facilitate overlapping of the 

20 guide image GP and the monitor image QP. The process can be 
selected in accordance with user' s desire. In the present 
embodiment, an example processed by texture mapping is shown. 

The attribute information AJ is header data 
accompanying the model data MD. The attribute information 

25 AJ gives the indication about from which position and how 

the object Q should be shot for inputting three-dimensional 
data of all-around surface or a predetermined area of the 
object Q. The attribute information AJ includes a shooting 
direction, a shooting angle, and the number of shots. The 

30 guide image GP is generated from the model data MD in 

2 4 




accordance with the attribute information AJ. 

Furthermore, the attribute information AJ can give an 
instruction of inputting with high resolution utilizing 
zooming function for a part such as eyes and a nose that 

5 require particularly detail data when the object Q is a 

human head. The attribute information AJ can also give an 
instruction of inputting with low resolution, i. e. , with 
less data for a part such as a cheek or a head whose shape 
does not change so rapidly. 

10 According to the attribute information AJ, the model 

data MD after process such as the wire frame are displayed 
on the finder 21 as a predetermined guide image GP. 
Hereinafter, if it is necessary to distinguish the guide 
image GP from others by its displayed shape, a suffix 

15 number is added in such way as "GP1" or "GP2. " 
[The first input method] 

The method for inputting the three-dimensional data 
using the monitor image QP and the guide image GP1 will be 
explained with reference to Fig. 8 and Figs. 9A-9C. 

20 First, the user selects the model data MD corresponding 

to the kind of the object Q to be inputted. The selected 
model data MD are read out of the model data memory 75 in 
accordance with the instruction by the system controller 61 
and are displayed on the finder 21 as the guide image GP1. 

25 in Fig. 9A, the guide image GP1 is instructing the user 

to shoot the object Q from the front. The user does 
framing of the object Q so that the monitor image QP is 
overlapped on the guide image GP exactly. As shown in Fig. 
4, either the three-dimensional camera 2 or the object Q can 

30 be moved for framing. By performing zooming or by moving 
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the three-dimensional camera 2, various resolutions can be 
used for inputting. For example, a high resolution is used 
for inputting detail data required to a portion such as 
eyes and a nose, while a low resolution can be used for 
5 inputting a portion such as a cheek and a head whose shape 
is not changed so rapidly. Thus, unnecessary increase of 
data can be avoided. After overlapping, the shutter button 
27 is pushed so that the first three-dimensional data are 
inputted. 

10 After inputting the first three-dimensional data, the 

guide image GP1 is changed to the guide image GP2. The 
guide image GP2 gives the instruction about from which 
position the object Q should be shot the next time. The 
next shooting is performed in accordance with this 

15 instruction so that the registration can be performed with 
high accuracy. 

In Fig. 9B, the guide image GP2 instructs to shoot the 
object Q obliquely from the left direction next. The user 
does the framing of the object Q so that the monitor image 

20 qp is overlapped on the guide image GP2. Namely, the 

relative position of the object Q to the three-dimensional 
camera 2 is changed so that the monitor image QP overlaps 
the guide image GP2. When the monitor image QP agrees the 
guide image GP2, the shutter button 27 is pushed for 

25 inputting the second three-dimensional data. In the same 

way, the positions for shooting are sequentially instructed 
to the user by the guide image GP. The user performs the 
inputting sequentially in accordance with each instruction 
of the guide image GP. 

30 i n Fig. 9A-9C, the guide image GP gives only the 
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instruction about from which direction the head portion 
should be shot. However, another instruction can be given 
about inputting the detail data of eyes and a nose with 
high accuracy by displaying the enlarged guide image GP 
5 thereof. 

Thus, the sequential instruction of the guide image GP 
can be given automatically by program. Furthermore, instead 
of the automatic sequential instruction, the contents of 
the attribute information AJ can be selected manually and 

10 the guide image GP is displayed in accordance with the 
selected order. 

The inputting on the basis of the guide image GP makes 
it possible to input the three-dimensional data of the 
object easily, quickly, and correctly. Furthermore, the 

15 registration of the plural sets of shot three-dimensional 
data can be performed easily and precisely, so the composed 
image of the object for all-around surface or a 
predetermined area can be obtained easily. Furthermore, the 
composed image without omission can be obtained easily by 

20 setting the attribute information AJ properly so that all 
of the necessary portions of the object are shot. 

If the composed image has a lack point, the data can be 
compensated by interpolation process. In this case, the 
first generated model data MD, the corrected model data MD 

25 or the three-dimensional data obtained by the previous 
shooting can be used. 

Furthermore, the peripheral portion of the inputted 
three-dimensional data whose reliability is low can be 
omitted from use and display. 

30 Fig. 10 is a flowchart showing an operation and a 
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process of inputting the three-dimensional data in a first 
inputting method. 

The currently captured object Q is displayed on the 
finder 21 as the monitor image QP (#1). The guide image 

5 GP1 corresponding to the object Q is displayed on the finder 
21 (#2). The framing is performed so that the monitor 
image QP is overlapped on the guide image GP1 (#3). The 
view point position for the finder display is determined on 
the coordinate system of the guide image GP1 displayed on 

10 the finder 21 (#4). The view point at this time is 

determined as "the first view point. " The first three- 
dimensional data are inputted from the first view point (#5). 

The next guide image GP2 is displayed (#6). The 
framing is performed so as to overlap the displayed guide 

15 image GP2 (#7), and the second three-dimensional data are 
inputted (#8). A parameter for converting the coordinate 
system of the second three-dimensional data into the 
coordinate system of the first three-dimensional data is 
determined (#9). Namely, the three-dimensional data 

20 inputted with overlapping on the finder 21 are data of each 
coordinate system, so the conversion between the coordinate 
systems is necessary. The registration of the three- 
dimensional data means a conversion into one of the 
coordinate systems. The method of the registration is the 

25 same as explained in the first embodiment. The coordinate 

conversion of the second three-dimensional data is performed 
using the determined parameter (#10), and the initial 
registration is completed (#11). A fine adjustment is 
performed by ICP process for further registration with high 

30 accuracy (#12). 
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The movement quantity of each instruction of the guide 
image GP is predetermined by the attribute information PD. 
The relative movement quantity of the three-dimensional 
camera 2 to the object Q is determined by the overlapping. 

5 The plural sets of three-dimensional data inputted from 

plural positions are converted into data of the coordinate 
system of the first shooting in accordance with the 
movement quantity. Thus, the registration can be performed. 
As explained above, the three-dimensional data inputted 

1° from plural positions are inputted with high accuracy of 
registration by overlapping the monitor image QP on the 
guide image GP. Therefore, a post registration is not 
required. 

Thus, the host 3 performs pasting process on the basis 

15 of the three-dimensional data from plural positions with 
high accuracy of registration. Therefore, the pasting 
process in the host 3 can be performed in high speed and 
high accuracy. 
[The second input method] 

20 Even if the kind of the object Q to be inputted is the 

same, the shape thereof can be a little different. For 
example, each human head has its unique feature such as 
round, long or protruding back. Furthermore, a head of an 
adult is different in size from that of a child. As a 

25 result, the prepared guide image GP can be substantially 
different from the shape or the size of the object Q, so 
that the overlapping cannot be performed properly. 

In the second input method, the model data MD is 
corrected by matching the guide image GP with the monitor 

30 image QP, and the scale adjustment is performed so that the 
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scale of the monitor image QP and the scale of the guide 
image GP are matched with each other. 

Fig. 11 is a diagram showing a monitor image QP and a 
guide image GP in a second inputting method. 

5 In Fig. 11, the guide image GP3 that is larger than the 

monitor image QP in the vertical direction is displayed on 
the finder 21. First, the user does framing of the object Q 
so that the monitor image QP is overlapped substantially on 
the guide image GP3. Next, matching of the guide image GP3 

10 with the monitor image QP is performed. Points such as a 
parietal portion PI, an end of a chin P2 and a base of an 
ear P3 are given to the model data MD as the attribute 
information AJ, and a point on the monitor image QP 
corresponding to each of the points is designated. Another 

15 method can be adopted, in which any corresponding point is 
designated. 

Then, the ratio of the movement quantity in the 
vertical direction and that in the horizontal direction 
between the matched points is averaged. The scale change 

20 is performed for the entire of the model data MD, i. e. , the 
guide image GP. Thus, the guide image GP4 after scale 
adjustment of the aspect ratio (not shown) is generated. 

Then, the framing of the object Q is performed so that 
the monitor image QP is overlapped on the guide image GP4 

25 for inputting the first three-dimensional data. After 

inputting the first three-dimensional data, the next guide 
image GP is displayed in the same way as in the first input 
method. In the same way as in the input of the first three- 
dimensional data, the rough overlapping, the scale 

30 adjustment by the matching and the input of the three- 
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dimensional data are repeated in accordance with the 
instruction of the guide image GP. 

Thus, since the three-dimensional data are inputted in 
accordance with the instruction of the guide image GP in 

5 which the scale is adjusted to the shape of the object Q, 
the input is performed with high accuracy of the 
registration. Therefore, even if the shape of the object Q 
is a little different from others, a composed image with 
high accuracy can be obtained. 

10 in the same way as in the first input method, if the 

composed image has a lack point, the data can be compensated 
by interpolation process. Furthermore, the peripheral 
portion of the inputted three-dimensional data whose 
reliability is low can be omitted from use and display. 

15 Fig. 12 is a flowchart showing an operation and a 

process of inputting the three-dimensional data in the 
second inputting method. 

On the finder 21, the currently captured object Q is 
displayed on the finder 21 as the monitor image GP (#21). 

20 The guide image GP3 corresponding to object Q is displayed 

(#22). The framing is performed so that the guide image GP3 
is substantially overlapped on the monitor image QP (#23). 

The matching of the guide image GP3 and the monitor image 
QP is performed. The scale adjustment of the aspect ratio 

25 of the model data MD and the object Q is performed by 
designating corresponding points (#24). The view point 
position of the finder display is determined in the 
coordinate system of this position (#25). The view point at 
this time is determined as "the first view point." The 

30 first three-dimensional data are inputted from the first 
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view point (#26). A calibration is performed for the scale 
adjustment of the real size of the object Q and the model 
data MD (#27). 

The next guide image GP4 is displayed (#28). The 

5 framing is performed so that the guide image GP4 is 

substantially overlapped on the monitor image QP (#29), and 
the scale adjustment of the aspect ratio of the model data 
MD and the object Q is performed by designating 
corresponding points (#30). The second three-dimensional 

10 data are inputted by overlapping the monitor image QP on the 
guide image GP5 (not shown) after the scale adjustment 
exactly (#31). A parameter for converting the coordinate 
system of the second three-dimensional data into the 
coordinate system of the first three-dimensional data is 

15 determined (#32). A coordinate conversion of the second 
three-dimensional data is performed using the determined 
parameter (#33). In the same way as in the first input 
method, the initial registration is completed (#34) and a 
fine adjustment is performed by ICP process for further 

20 registration with high accuracy (#35). 

The step 24 and the step 30 mentioned above correspond 
to the matching means and the scale agreement means. 

In the above-explained embodiment, the guide image GP 
and the monitor image QP are displayed on the finder 21 

25 that is integrated to the three-dimensional camera 2. 

However, they can be displayed on a finder separated from 
the three-dimensional camera 2 or on a display 3b of the 
host 3. 

In the above-explained embodiment, a see-through 
30 display device utilizing a half mirror can be used as the 
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finder 21. In this case, power consumption of the three- 
dimensional camera 2 can be reduced. 

In the above-explained embodiment, the configuration, 
the shape, the position, the circuit and the process of 
each part or the whole of the measurement system 1 and the 
three-dimensional camera 2 can be changed in accordance with 
the tenor of the present invention. 

According to the present invention, the three- 
dimensional data of the object can be inputted easily, 
quickly and correctly. The composed image of the all-around 
surface or a predetermined area of the object can be 
obtained easily without any omission. 
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